Study on Building a High-Quality Homepage Collection from the Web Considering Page Group Structures

This disseration is devoted to investigate the method for building a high-quality homepage collection from the web efficiently by considering the page group structures. We mainly investigate in researchers’ homepages and homepages of other categories partly. A web page collection with a guaranteed high quality (i.e., high recall and high precision) is required for implementing high quality web-based information services. Building such a collection demands a large amount of human work, however, because of the diversity, vastness and sparseness of web pages. Even though many researchers have investigated methods for searching and classifying web pages, etc., most of the methods are best-effort types and pay no attention to quality assurance. We are therefore investigating a method for building a homepage collection efficiently while assuring a given high quality, with the expectation that the investigated method can be applicable to the collection of various categories of homepages. This dissertation consists of seven chapters. Chapter 1 gives the introduction, and Chapter 2 presents the related work. Chapter 3 describes the objectives, the overall performance goal of the investigated system, and the scheme of the system. Chapters 4 and 5 discuss the two parts of our two-step-processing method in detail respectively. Chapter 6 discusses the method for reducing the processing cost of the system, and Chapter 7 concludes the dissertation by summarizing it and discussing future work. Chapter 3, taking into account the enormous size of the real web, introduces a two-step-processing method comprising rough filtering and accurate classification. The former is for narrowing down the amount of candidate pages efficiently with the required high recall and the latter is for accurately classifying the candidate pages into three classes—assured positive, assured negative, and uncertain—while assuring the required recall and precision.